The TUNA-REG Challenge 2009: Overview and Evaluation Results
نویسندگان
چکیده
The TUNA-REG’09 Challenge was one of the shared-task evaluation competitions at Generation Challenges 2009. TUNAREG’09 used data from the TUNA Corpus of paired representations of entities and human-authored referring expressions. The shared task was to create systems that generate referring expressions for entities given representations of sets of entities and their properties. Four teams submitted six systems to TUNAREG’09. We evaluated the six systems and two sets of human-authored referring expressions using several automatic intrinsic measures, a human-assessed intrinsic evaluation and a human task performance experiment. This report describes the TUNAREG task and the evaluation methods used, and presents the evaluation results.
منابع مشابه
The TUNA Challenge 2008: Overview and Evaluation Results
The TUNA Challenge was a set of three shared tasks at REG’08, all of which used data from the TUNA Corpus. The three tasks covered attribute selection for referring expressions (TUNA-AS), realisation (TUNA-R) and end-toend referring expression generation (TUNAREG). 8 teams submitted a total of 33 systems to the three tasks, with an additional submission to the Open Track. The evaluation used a ...
متن کاملIntroducing Shared Task Evaluation to NLG The TUNA Shared Task Evaluation Challenges
Shared Task Evaluation Challenges (stecs) have only recently begun in the field of nlg. The tuna stecs, which focused on Referring Expression Generation (reg), have been part of this development since its inception. This chapter looks back on the experience of organising the three tuna Challenges, which came to an end in 2009. While we discuss the role of the stecs in yielding a substantial bod...
متن کاملTemplate-Based Surface Realization for Referring Expression Generation
This paper describes the template-based surface realizer which was used by the GRAPH team in the TUNA-REG Challenge 2009 in combination with an attribute selection algorithm used in previous editions of the TUNA challenges. The new template-based realizer is compared to the realizer used by the GRAPH team in the REG 2008 Challenge and to an improved version of the 2008 realizer. Results show no...
متن کاملXML Format Guidelines for the TUNA Corpus
This document forms part of the 2008 distribution of the TUNA Corpus, Version 1.0. This is the first public release of the complete TUNA Corpus of Referring Expressions. A subset of the corpus was used in the first Shared Task and Evaluation Challenge for NLG, the Attribute Selection for the Generation of Referring Expressions Challenge (ASGRE), co-located with the Workshop on Using Corpora in ...
متن کاملThe D-TUNA Corpus: A Dutch Dataset for the Evaluation of Referring Expression Generation Algorithms
In this paper, we present the D-TUNA corpus, which is the first semantically annotated corpus of referring expressions in Dutch. Its primary function is to evaluate and improve the performance of REG algorithms. Such algorithms are computational models that automatically generate referring expressions by computing how a specific target can be identified to an addressee by distinguishing it from...
متن کامل